An approach to measuring and assessing dependability for critical software systems

Traditional software testing methods combined with probabilistic models cannot measure and assess dependability for software that requires very high reliability (failure rate < 10 /hour) and availability (>0.999999). This 6 paper proposes a novel approach, drawing on findings and methods that have been described individually but have never been combined, applied in the late testing phase or early operational phase, to quantify dependability for a category of critical software with such high requirements. The concepts that are integrated are: operational profile, rare conditions, importance sampling, stress testing, and measurement-based dependability evaluation. In the approach, importance sampling is applied on the operational profile to guide the testing of critical operations of the software, thereby accelerating the occurrence of rare conditions which have been shown to be a leading cause of failure in critical systems. The failure rates measured in the testing are then transformed to those that would occur in the normal operation by the likelihood ratio function of the importance sampling theory, and finally dependability for the tested software system is evaluated by using measurement-based dependability modeling techniques. When the acceleration factor is large (over 100), which is typical for a category of software of interest, it is possible to quantify a very high reliability or availability in a reasonable test duration. Some feasible methods to implement the approach are discussed based on real data.